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BACKGROUND 



The study described in this report is part of an extensive program of research in the 
area of instructional strategies and decision models. This research program was accom 
plished over a five-year period in a computer -administered instructional (CAI) environ 
ment. The principal sponsor of HumRRO's CAI research and development efforts during 
that time was the Department of the Army— first, with the U.S. Army Research Office as 
the monitor under an overall program entitled Project IMPACT, then, with the U.S. 
Army Research Institute for the Behavioral and Social Sciences as monitor under Project 
CATALIST. A National Science Foundation (NSF) grant augmented the Army*s sponsor- 
ship of this research.^ 



PURPOSE 

The study described in this report was designed to test effects of learner control at 
the level of instructional sequencing within a self contained tutorial course, administered 
by an adaptive computet program. Applicability to other levels is speculative and clearly 
requires further research. 

The experimentation has as independent variables four features of our CAI system 
that afford the student a specified degree of control over the sequencing of instructional 
material. Three of the variables are student options that control remedial activity and 
acceleration. The fourth variable allows control over sequencing of topics at specified 
points in the course. The purpose is to assess the relative contributions and interactions 
of these variablts with respect to iiistructional effectiveness and efficiency. 



METHOD 

Following an entry test period, students were administered tutorial CAI instruction, 
a COBOL course (an average of 30 hours long), with four possible types of learner- 
control variables. These students were assigned at random to one. of 2^ factorial treat- 
ment conditions. Sessions were approximately three hours long per day with breaks left 
up to the individual. Following the instructional period, students were administered an 
'*exit questionnaire" covering their opinions about course aumuilstration, content, and 
instructional environment. 

During the conduct of the experiment, three types of measures were taken on each 
student, (a) entry characteristics, including information processing (Guilford*s Structure 
of Intellect), affective, and biographical data, (b) learner strategies, including type and 
frequency of control usage and the circumstances of their use, and (c) achievement and 
other performance-related measures including quiz scores, transit times, programming 
errors, opinions of topics, and Level of Aspiration (LOA) prior to the quiz of each topic. 



'NSF Grant GJ 774 to Human Resources Research Organization, "Research on Instructional 
Decision Models," Robert J. Seidel, Principal Investigator. See Research on Instructional Djcision Models, 
HumRRO Final Report FR-Dl-73-6, December 1973. 



1 



Assessments were made of the relative contributions and interactions among the learner- 
control and entry characteristic variables with respect to instructional effectiveness and 
efficiency as represented by the dependent measures. 



RESULTS AND IMPLICATIONS 

B'lrst, the study developed a well-tested instructional vehicle that meets the criterioi, 
of student mastery, a prerequisite for valid research in an instructional environment. 

Secondlv, the study was performed in a rich instructional environment, preferred for 
generalizmg results of Aptitude by Treatment Interaction (ATI) studies to a real instruc- 
tional world. 

The third significant aspect of the current study has been the development of a very 
useful means to characterize high and low performers with an operationally defined set of 
criteria that has highlighted the value of discriminant function analyses in instructional 
resewch settings. Of great importance is the finding that the particular individuals 
designated high or low performers differed depending upon the particular instructional 
tasks. Yet, the phenomenon of high and low performance was consistent across two 
divisions of the course. High and low performe o differed with respect to the usage of 
options, as well as their Level of Aspiration settings concerning their performance % 
Research is needed to identify more specifically the taxonomic characteristics of instruc- 
tional tasks related to student profiles of high and low performers. 

Another significant finding in our study was that self -assessment can make a 
significant contribution to instructional management, whether the latter be by students or 
by the learning svstem. The next step that should be taken is to use the instructional 
options based on expectations, as part of the decision -making process in an adaptive 
instructional environment. A proposed prescriptive use of LOA as an Expectancy 
Operator is described. Lastly, research requirements for systematic study of levels and 
types of self-managed learning paradigms are discussed. 



PREFACE 



This report describes ^he research performed in Work Unit CATALIST in the area of 
instructional strategies and decision models. The specific study presented in this report 
was designed to test the effects of learner control of instructional sequencing within a 
tutorial computer-administered instruction (CAI) course. 

The work was conducted at HumRRO Eastern Division, Alexandria, V'vginia, 
Dr. J. Daniel Lyons, Director. Dr. Robert J. Seidel is the Program Director for Work Unit 
CATALIST. In addition to the authors of this report, contributors to the research 
activities have been Dr. Edward Schneider, Judy Compton, and William Underhill. 
Assistance was provided in data collection and analysis by SP4Jack Huber and 
SP4 Gerald Grady while they were stationed at HumRRO. 

HumRRO research for the Department of the Army under Work Unit CATALIST is 
performed under Army Contract DAHC19-73-C-0004. Computer-administered instructitn 
research is conducted under Array Project 2Q763731A734. A National Science Founda- 
tion Grant GJ 774 to the Human Resources Research Organization, "Research on 
Instructional Decision Modcjls," Robert J. Seidel, Principal Investigator, augmented the 
Army's sponsorship of this research. The CATALIST work is conducted under the 
sponsorship of the U.S. Army Research Institute for the Behavioral and Social Sciences, 
with Dr. Joseph S. Ward serving as the technical monitor. 



Meredith P. Crawford 
President 

Human Resources Research Organization 
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Learner Control of 

Instructional Sequencing Within an 

Adaptive Tutorial CM Environment 



chapter 1 
INTRODUCTION 



BACKGROUND 

The study described in this report is part of an extensive program of research in the 
area of instructional strategies and decision models. This research program was accom^ 
plished over a five-year period in a computer-administered instructional (CAI) environ- 
ment. The principal sponsor of HumRRO's CAI research and development efforts during 
that time was the Department of the Army. First, the U.S. Army Research Office was 
the technical monitor for an overall program entitled Project IMPACT. Then, the U.S. 
Army Institute for the Behavioral and Social Sciences was the monitor for Project 
CATALIST. A National Science Foundation (NSF) grant augmented the Army's sponsor- 
ship of this research.^ 



RATIONALE 

The problen of student-controlled learning has . jeived a great deal of attention 
recently. It is obviously of prime importance in individualized instruction. Relevant to 
the study of student-controlled instructional strategies is the whole area of discovery 
learning (Gagne^ ). Some learning studies have indicated a high degree of mastery by using 
discovery techniques (e.g., Gagne and Bassler^ ). 

But what are the psychological dimensions involved? Many assertions have been 
made (Bushnell*, Hartley, Sleeman, etaL'^) that learner-controlled instruction can over- 
come the lack of predetermined, explicit models of instructional processes. But little 
systematic exploration of the nature and degree of desirable learner-generated control 
processes in an adaptive teaching system has occurred. Grubb^ has demonstrated the 
general feasibility of teaching statistics with learner control. Swets'' found no difference 
between learner or program control in learning to identify complex auditory stimuli. 
Hartley,* using learning of logarithms as the problem-solving task, also found no 

*NSF Grant GJ 774 to Human Resources Research Organization, "Research on Instructional 
Decision Models," Robert J. Seidel, Principal Investigator. See Research on Instructional Decision Models, 
HumRRO Final Report FR-Dl-73-6, December 1973. 

^R.M. Gagne. Condition$ of Human Learning, Holt, Rinehart and Winston. New York, 1965. 

^R.M. Gagne and O.C. Bassler. "'Study of Retention of Some Topics of Elementary Non-metric 
Geometry,** Journa/ of Educational Psychology, 1963. 

^D.D. Bushnell. "For Each Student a Teacher," Science, July 1966. 

^ J.R. Hartley, D.H. Sleeman, et al 'Teaching and Evaluation Logics in a Computer Assisted Learning 
System,** presentation at University of Leeds, England, September 1969. 

*R.E. Grubb. "Learner Controlled Statistics," Research Report, IBM Corporation, Los Gatos, 
California, April 1967. 

''j. Swets, et ai "Further Experiments on Computer-Aided Learning of Sound Identification and 
Cope Light Pen," NAVTRADEVCEN Technical Report 789 2, Bolt, Beranek and Newman, Inc . April 1964 

•j.R. Hartley. "Optional and Controlled Branching. Comparison Studies," The Journal of Pro 
grammed Instruction, vol. 3, No. 4, 1966. 
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differences (although his data are confounded by possible ceiling effects). Mager^s 
research/ on the other hand, indicated significant learning value in having subjects 
determine sequencing of materials. 

CarLonnel and Collins^ and Grignetti, et ai'^ constructed an innovative CAI system 
called SCHOLAR. Its prime instructional feature is a mixed initiative mode of interaction 
with students. In this mode, the student can access information about a topic by typing 
questions and the system can direct the student's efforts by asking questions of its own. 
Unfortunately, no data except for sample protocols have as yet been published on 
student residence time in the various modes or on the effectiveness or efficiency of 
mixed-initiative versus conventional strategies. 

In a carefully structured study by Olivier,* the student group having the capability 
for self-selection of instruction, did significantly worse than other system controlled 
counterparts on criterion tests. Generalized interpretation, however, is difficult because of 
the rather short instructional period of 30 15 minutes, which could have had some kind 
of a warm-up effect. Similarly, Fry' found suggestive hints that the value of learner 
control is affected by individual difference variables, however, the instruction was 
relatively brief (two one-hour sessions) and the data inconclusive. 

When these findings are compared with those of Grubb,^ Mager,'' and Judd, 
Bunderson, and Bessent,* hard and fast conclusions concerning the value of student- 
versus system-controlled instruction are impossible. In these other studies, more than just 
learner-controlled sequencing is involved. Moreover, the individual student's profile by 
treatment interaction was considered in only three of the experiments (Fry, Olivier, and 
Judd, et ai). Even in these cases the aptitude measures were of a general nature and not 
necessarily the best task-relatable content for the instruction given. Research is needed to 
clarify this issue.^ 

The key to optimal allocation of learner controls in the instructional decision 
process would be for basic research in human learning to (a) identify those components 
of strategy selection and use of which students are capable, (b) relate these components 
to individual characteristics, and (c) determine where program control can or cannot 



' R.F. Mager. "Learner-Controlled Instruction-1958-1964/* Programmed Instruction, vol. 4, no. 2, 

1964. 

^J.R. Carbonnel and A.M. Collins. "Mixed Initiative Systems for Training and Decision Aid 
Applications," Hanscomb AFB~ Electronic Systems Division ESD TR 70 373, Bedford. Massachusetts, 
November 1970, 

^Mano C. Gngnetti, Laura Gould, Catherine L. Hausmann, Alan G. Bell, Gregory Harris, Joseph 
Passafiume. -Mixed-Initiative Tutorial S>stem to Aid Users of the On Line System (NLS),'* Hanscomb 
AFB-Electronic Systems Division ESD-TR-75-58, Bedford, Massachusetts, November 1974. 

'^W.P. Olivier. '*Learner and Program-Controlled Sequences of Computer Assisted Instruction,** 
oaper presented at the annual meeting of the American Educational Research Association, New York, 
February 1971. 

'^J.P. Fry. ''Interactive Relationship Between Inqulsitiveness and Student Control of Instruction," 
Journal of Educational Psychology, vol.63, no. 5, October 1972, issued as HumRRO Professional 
Paper 22-72, December 1972. 

^Grubb, op. cit. 

^ Mager » op. cit, 

*W.A. Judd, C.V. Bunderson, and E.W. Bessent. "An Investigation of the Effects of Learner 
Control in Computer-Assisted Instruction Prerequisite Mathematics (MATHS),** Technical Report No. 5, 
Computer-Assisted Instruction Laboratory, The University of Texas at Austin, 1970. 

**See. Albert Hickey. "Research Guidelines for Computer Assisted Instruction,** presentation at 
Association for the Development of Computer Based Instructional Systems (ADCIS) Winter Meeting. 
Washington, D.C., January 29-February U 1974. 
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handle the same components (Seidel* ). The premise is intended to imply that an 
all-or-none conclusion for locus of control is highly unlikely. We could then arrive at a 
cost-effective justification for optimally allocating components of instructional decision 
making to individual students or programs in an adaptive teaching system. Both Pask*s 
cybernetic approach and that of the author and his associates (Seidel, et ai^ , SeideP , 
Seidel and Kopstein^) are directed toward this end. 

As an empirical example of this approach, the present experiment is unique in three 
respects: 

(1) It employs a combined manipulative-correlational technique in order to 
further the understanding of the relationship between individual student characteristics 
and effectiveness of self-management or learner control. 

(2) It is one of the few studies involving a rich, contingency -structured instruc- 
tional environment. 

(3) A battery of test instruments was administered to develop comprehensive 
profiles for individual students including cognitive, motivational, and specific performance 
indicators relevant to the instruction. 



* R.J. Scide). 'Theories and Strategies Related to Measurement in Individualized Instruction/* 
HumRRO Professional Paper 2-71, March 1971. 

^ R.J. Seidel, et ai **Project IMPACT. Computer Administered Instruction Concepts and Initial 
Development,** HumRRO Technical Report 69-3, March 1969a. 

^R.J. Seidel, et al **Project IMPACT. Description of Learning and Prescription for Instruction," 
HumRRO Professional Paper 22-69, June 1969b. 

R.J. Seidel. "Computers in Education. The Copernican Revolution in Education Systems,*' 
Computers and Automation, March 1969, issued as HumRRO Professional Paper 16 69, May 1969. 

^ R.J. Seidel and F. Kopstem. "A General Systems Approach to the Development and Maintenance 
of Optimal Learning Conditions," HumRRO Professional Paper 1-68, January 1968. 
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METHOD 



OVERVIEW OF EXPERIMENTAL VARIATIONS 

Before describing the experimentd design in detail, it is important to emphasize that 
student-centered decision making can take place on many levels in education. It may 
involve a choice of degree program, scheduling, objectives, strategy for studying, or use of 
resources. The present study was designed to test effects at the level of instructional 
sequencing within a self-contained tutorial course administered by an adaptive computer 
program. Applicability to other levels is speculative and requires further research. 

Four features of the CAI system that afford the student a specified degree of 
control over the sequencing of instructional material serve as independent variables in the 
experiment. Three of the variables are student options that control remedial activity and 
acceleration. If the student types REVIEW, he returns to the beginning of the topic. If 
the student types RECAP, he is shown a list of topics that he has previously completed 
and is allowed to choose one for review. If the student types QUIZ, he is branched from 
his current location in a topic to the beginning of the "Quiz" Section of that topic. 

The fourth variable, ROUTE, allows student control over sequencing of topics at 
specified points in the course. The system makes the options available, and the student 
uses tnem or not, as he wishes. In topic sequencing, the student must choose the next 
topic, if the choice is available. 

Following a pretest period, students were administered tutorial CAI instruction with 
four possible types of learner-control variables. These students were randomly assigned to 
one of the 16 conditions in the (2^) factorial design. Sessions were approximately three 
hours long per day with breaks left up to the individual. Following the instructional 
period, students were administered an "exit questionnaire" covering their opinions about 
course administration, content, and instructional environment. 

During the conduct of the experiment, three types of measures were taken on each 
student, (a) entry characteristics, including aptitude, affective, and biographical data; 
(b) learner strategies, including type and frequency of control usage aiid the circum- 
stances of their use, and (c) achievement and other performance-related measures 
including quiz scores, transit times, programming errors, opinions of topics, and Level of 
Aspiration (LOA) prior to the qui*; on each topic. 

Thus, the purpose of the present experiment was to assess the relative contributions 
and interactions among the learner-control, entry charactensilcs, and dynamic motiva- 
tional variables, with respect to instructional effectiveness and efficiency, as represented 
by the dependent measures. 

DESCRIPTION OF ENTRY CHARACTERISTICS MEASURES 

Student entry characteristics were measured in these four areas. Structure of Intel- 
lect Factors, Motivation, Computer Programming Aptitude, and Reading Comprehension. 

The Entry Characteristics Test (ECT) battery consists of 27 instruments (26 are time 
tests) that yield 35 distinct scores. Their testing time ranges from 2 to 70 minutes, with a 
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majority of the tests taking less than 20 minutes. In general, the tests are of the 
paper-and-pGncil variety and designed for administration in group testing sessions. 

Structure of Intellect Factor Tests 

These tests measure J.0 factors and were used most recently by Bunderson.* There 
are 27 tests used for factor measurement. A description of the factors and the tests used 
to measure them follows: 

(1) General Reasoning (Two tests). This factor has been described as "the 
ability to solve a broad range of reasoning problems, including those of a mathematical 
nature" (French, Ekstrom, and Price'). The tests selected to define this factor are the 
Ship Destination Test (Sheridan Psychological Services, Inc.) and Necessary Arithmetic 
Operations (Educational Testing Service). 

(2) Induction (Three tests). This factor has been described as "associated 
abilities involved in the finding of general concepts that will fit sets of data, the forming 
and trying out of hypotheses" (French, et a/.). The Letter Sets Test (Educational Testing 
Service), Locations Test (Educational Testing Service), and Figure Classification Test 
(Educational Testing Service) define this factor. 

(3) Figural Adaptive Flexibility (Five tests). French, et aL describe this factor 
as "the ability to change set in order to meet new requirements imposed by figural 
problems." The following five tests define this factor: 

(a) Match Problems IV— Parts 1 and 2 (Aptitudes Research Project, 
University of Southern California). 

(b) Match Problems V (Educational Testing Service). 

(c) Word Codin'" Test (designed by Lennart Sjoberg, John Frederiksen, 
and Victor . anderson). 

(d) Decoding Test (designed by Lennart Sjobei^, John Frederiksen, and 
Victor Bunderson). 

(4) Verbal Reasoning (Three tests). This factor has been given a number of 
different names, including "Deduction" by Thurstone, "Logical Reasoning" by Guilford, 
and "Syllogistic Reasoning" by French, etaL^ French, etaL describe it as "ability to 
reason from stated premises to their necessary conclusions." The Nonsense Syllogisms 
Test (Educational Testing Service), Logical Reasoning Test (Sheridan Psychological 
Services, Inc.), and Inference Test (Educational Testing Service) were selected to define 
this factor. 

(5) Symbol Substitution (One test). Guilford and Hoepfner* classify this factor 
as convergent production and define it as "the ability to produce a completely 
determined, symbolic deduction from given symbolic information, where such an implica- 
tion has not been practiced, as such." One test defines this factor: Sign Changes 
(Aptitudes Research Project, University of Southern California). 

(6) Chunking Memory (One test). This is a new factor postulated by 
Bunderson, who has designed two tests. In the present study only one, the Binary Digit 
Span Test, measures this factor. 

(7) Memory Span (Two tests). This factor has been described as "the ability to 
recall perfectly for immediate reproduction a series of items after only one presentation 

* C.V. Bunderson. Transfer of Mental Abilities at Different Stages of Practice in the Solution of 
Concept Problems, Research Bulletin RB 67 20, Educational Testing Service, Princeton, New Jersey, 1967. 

' J.W. French, R.B. Ekstrom, and L.A. Price. Kit of Reference Tests for Cognitive Factors, Educa- 
tional Testing Service, Princeton, New Jersey, 1963. 

^Ibid. 

^J.P. Guilford and R. Hoepfner. "Structure of Intellect Tests and Factors," Reports from the 
Psychological Laboratory, no. 36, University of Southern California, Los Angeles, 1966. 



of the series*' (French, etai^ ). The marker tesUs for this factor are the Auditory Number 
Span Test and the Auditory Letter Span Test (both from Educational Testing Service). 

(8) Associative Memory (Three tests). This factor is defined as the ability to 
remember paired associates. The three tests that define this factor are: the Picture- 
Number Test, the Object-Number Test, and the First and Last Names Test (all from the 
Educational Testing Service). 

(9) Perceptual Speed (One test). This factor is described as the ability to make 
comparisons and find figures fast and accurately. The test that defines this factor is the 
Number Comparison Test (Educational Testing Service). 

(10) Spatial Scanning (One test). This factor is defined as "speed in visually 
exploring a wide or complicated spatial field" (French, et c/.). The Maze Tracing Speed 
Test (Educational Testing Service) measures it. 

Motivation 

Included in the Entry Characteristics Test battery are those tests that were selected 
to measure anxiety and achievement motivation. The psychological literature is replete 
with studies showing relationships between anxiety and learning in laboratory situations 
(Spence and Spence' ). Recent studies by Hansen and associates^ have also shown some 
value in the study of anxiety as it relates to performance in CAL The tests in the ECT 
battery are the IP AT Anxiety Scale Questionnaire (Institute for Personality and Ability 
Testing), the Sarason Task Anxiety Questionnaire (adapted from Mandler and Sarason"* ), 
and the Sentence Completion Test of Achievement Values (Mukherjee^ ). 

Computer Programming Aptitude 

A survey of the literature showed that the most widely used tests of aptitude for 
programming have been IBM's Programmer Aptitude Test (PAT) and Revised Programmer 
Aptitude Test (RPAT). A large body of reliability and validity data is associated with 
these tests. Recently, the PAT and RPAT have been replaced by the Aptitude Test for 
Programmer Personnel (ATPP), which is included in the battery. This test correlates 
highly with both PAT and RPAT. 

A second test. Primary Mental Abilities (PMA), is used in the ECT battery to 
measure programmer aptitude. The Primary Mental Abilities Test has been used for years 
by the RAND and System Development Corporations as a programmer selection device 
(Perry and Cantley^ ; Rowan'' ). 

The Army uses a programming aptitude test developed by a civilian company that is 
very similar to the ATPP. While the test is not included in our battery, the scores for 
military subjects were obtained for research purposes. 



* French, Ekstrom , and Price, op. ciL 

^K.W. Spence and J.T. Spence (eds.). The Psychology of Learning and Motivation.vol 1. Academic 
Press, New York, 1967. 

^ D.N. Hansen, W. Dick, and H.T. Lippert. "Annual Progress Report" (January 1, 1968--December 31, 
1968), Report No. 7, CAI Center. Institute of Human Learning, i'Morida Sta' University, Tallahassee, 1969. 

''G. Mandler and S.B. Sarason. "A Study of Anxiety and Learning Journal of Abnormal and 
Social Psychology, vol. 44, 1952, pp. 166-173. 

'^B.N. Mukherjee. Development of a Forced Choice Test of Achievement Motivation, Report Number 
CRP-S-113. Indiana University, Bioomington. Indiana. August 1964. 

*D.L. Perry and G. Cantley. Computer Programmer Selection and Training in System Development 
Corporation, Tech. Memo 2234. System Development Corporation, Santa Monica, California, 1965. 

^T.C. Rov\^an. "Psychological Tests and Selection of Computer Programmers.** hurnal of the 
Association for Computing Machinery, vol. 19, 1957. 
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Reading Comprehension 



The Entry Characteristics Test battery mdudi's ouv test of reading comprehension, 
the Reading Comprehension Cooperative Enghsh Test (Cooperative Test Division of 
Educational Testing Service). This instrument provides four scores; vocabulary, level of 
reading comprehension, speed of comprehension, and total reading comprehension 
(level + speed/2). 



EXPERIMENTAL COURSE 

The experimental course consisted of the first two divisions of a four-division course 
designed to teach COBOL programming.' The first two divisions comprised 22 modules 
and about 30 hours of instruction. 

Formative evaluation of the initial version of the course, COBOLl, was conducted 
with high school students and with Army students. Data were collected and analyzed on 
a total of 42 students. The evaluative data showed that subsets of instruction needed to 
be smaller for learning and research purposes. Thus arose the modular concept employed 
in C0B0L2, the second iteration COBOL course developed by HumRRO. Also, more 
opportunities for program writing and training in debugging procedures (through gaming) 
were provided for the student in C0B0L2. 

The Instructional Decision Model (IDM) and associated support software also under- 
went changes at this point, with a major element being the separation of the logic of the 
IDM from the instruction. This development was extremely useful because it permitted 
the IDM to be modified unconstrained by the course content. 

The C0B0L2 course and associated logic were debugged and evaluated with 83 
students of various civilian and military backgrounds. Following this, the 80 experimental 
subjects were administered C0B0L2. A total of 205 students have taken the various 
versions of the COBOL materials. 

The course structure (Figure 1) reflects the essential elements of a well-defined 
schema of objectives (i.e., single behavioral objectives are defined by single course 
objectives). The set of behavioral objectives for a division is sufficiently large and 
complex to require further partitioning into smaller subsets. This produces a Module 
Objective, operationally defined as the minimal observable subset of Terminal Achieve- 
ment Objectives. 

Modules are partitioned 'jito sections. The first is the A (Administrative) Section, 
which contains a variety of administrative documentation that the student does not see, 
for the most part, but that is used to provide course management information. The 
second is the T (Telling) Section in which relevant subject-matter information is 
presented to the student. The third is the P (Practice) Section which permits the student 
to practice objective-related behavior. The P-Section is followed by the Q (Quiz) Section 
which tests for achievement of the behavioral objective(s). A module can have several 
versions of each section. The limitation is, of course, that each version cover the same 
basic material and, therefore, teach toward the same objective. The differences between 
versions are differences of form and/or thinking rather than content. For example: 
Version 1 of a T-Section (T^) may require extensive reading, whereas Version 2 of the 



The COBOL course contained four divisions with a total of 33 topics comprising about 60 hours of 
instruction. A reduction in resources coupled with impending temporary loss of HumRRO *s in house 
computing facilities necessitated shortening the course in order to guarantee a sufficient number of 
subjects for the experiment. 
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T-Section (T2) for the same module may be highly pictorial or contain smaller chunks of 
information per display. 

An author must write a module so that it instructs toward one objective as that 
objective is represented in the prerequisite structure (Figure 2). That is, if a module 
(objective) does not have a linearly prerequisite module, the module being written must 
be independent of other modules. For example, in Division A, as represented in Figure 2, 
notice that Modules F, G, and H must each be written so that the author of any of these 
modules assumes only mastery of the objective taught by Module E when the student is 
taking Modules F or G or H. The author of Module I, however, must assume that the 
student has mastery of the objectives taught in Modules F and G and H. The prerequisite 
structure shown was established for the C0B0L2 course during its design phase and is 
based on an analysis of the relative nestings of behavioral objectives within others. 

The implications of this design for the ways a student can learn the materials is 
evident in Figure 3, a partial diagram of the path structure through Division B (if the 
student chooses Module C at his first choice point). Keeping in mind that the C0B0L2 
course is a self-contained tutorial (no non-system instructional support), the author must 
create the instruction so that all students, no matter what path they take to get to the 
module, can comprehend and thereby master any previously required module. For 
Module I this means all four modules in Division A, regardless of path chosen. 

The design of the course and the total CAI system at HumRRO also included special 
features that permitted the student or the system to exercise instructional strategies in 
the form of options for RECAP, REVIEW, ROUTE, PRACTICE, QUIZ, and INFO. All 
but the last of these options served as branching mechanisms for individualizing instruc- 
tion. The INFO option was designed for on-line glossary assistance with key terms and 
programming diagnostics. For purposes of ensuring the feasibility of the study, the 
PRACTICE option was not provided during the conduct of the research. 

Each student was told at the beginning of the course which options would be 
available to him. In the regular (non-experimental) version of C0B0L2, directions for 
exercising the options were included in the student reference manual. During the experi- 
ment, as a reminder, the particular options available were noted on a similar sheet placed 
over the cover of the student manual. An example of these directions is shown in 
Figure 4. 



STUDENT SUBJECTS 

Ninety percent of the sample (N=80) were paid volunteers recruited through adver- 
tisements in the local newspapers, the remaining subjects were military personnel, also 
volunteers, supplied by the Army's Project Transition— an activity designed to assist 
separating soldiers in developing job-related skills for civilian life. 

Our experience in the first COBOL course demonstrated that students who were 
severely deficient in programming aptitude were generally incapable of acquiring even the 
basic skills taught in the course as designed. Since such students provided little usable 
data while placing an additional strain on our already limited resources, we screened out 
any prospective student whose score on the IBM Programmer Aptitude Test fell below a 
raw score of 46 (a "low C") by more than one standard deviation. 

To make our experimental findings relevant to real world training, we intended that 
our subjects reflect the characteristics of programming trainees who are generally young, 
with a minimum of some high school, and naive with respect to programming. The data 
indicate that these requirements were met. Table 1 summarizes the relevant biographical 
characteristics of the sample. 



ERIC 



c 



A 



B D 



D 



B 
F + 



3 
3 



-[3: 



-Fx 



E G 



A 



Where: 

Modules within the 
brackets Q J appear 
together on a menu 
(or the student. 
In program paths 
• - exit points 
entrance points 
(a single module 
or a menu). 



E 



E 



.Ft 

"a" 



t--Cl:: 



F 



B- 

"e" 

.F. 

"a" 



E 



B 

"a" 



- E- 
E- 

"f"- 
"a"- 

J- 

"g"- 

_Aj- 
F- 





"a"^ 

G 



\\V 

\\\\\\ 
-G \\\\\\ 

-G.\^ \\\\\\ 



■-F \\\^ ^\\\ 

g-._'-'-<-;:n<=M 

-F 




-E - 
- E- 

a" 

G_ 
G 
F 

A"+ 
F 
G 
A 



"//',' , I 'lit 



■-G'/fn/l'l 
/ 1 II I III I, 

■-^//I'ln 

-^1 If 
- °/// 
-%'i 

— gV 



-A 



Figure 3. Partial Path Diagram for Division B. 
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REVIEW, QUIZ, and RECAP Options 



You may change the kind and amount of instruction you receive by using any of the options listed below. 
Remember that options are not available tr you during quizzes. 

To use an option, type it as an INFO-request. 

Use your light pen to move the cursor to the dash after INFO. Move the cursor one more space to the 
right by hitting the SPACE key one time. Immediately after the dash next to INFO , type the name of 
the option you wish to use. Make sure there is no extra space between the dash after INFO and the 
word you type. 

EXAMPLE. A request to RECAP should look like this: INFO-RECAP 

REVIEW - Goes back to the beginning of the explanation section. 
QUIZ - Skips all the remaining instruction before the quiz. 
RECAP - Allows return to any topic previously taken. 



Figure 4. Example of Directions Provided Experimental Subjects, 

Table 1 



Pertonal Data for Experimental Subjects 





Number of 






Responses in 


Percent of 


Characteristic 


Category 


Total Group 


Sex 






Male 


45 


56.3 


Female 


35 


43.7 


Age 






Less than 21 years 


22 


27,5 


21 -25 years 


28 


35.0 


26 • 30 years 


16 


20,0 


31 • 35 years 


3 


3.8 


36 - 40 years 


4 


5.0 


More than 40 years 


7 


8.8 


Vision Difficulties 






Yes 


46 


57,5 


No 


34 


42,5 


Typing Speed 






Non-typist 


24 


30.0 


1 . 20 WPM 


7 


8.8 


21 -40 WPM 


32 


40.0 


41 -60 WPM 


10 


12.5 


61 -80 WPM 


1 


1.2 



Greater than 80 WPM 2 2,5 

No response 4 5.0 
(Continued) — 
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Table 1 (Continued) 
Personal Data for Experimental Subjects 





Number of 






Responses in 


Percent of 


Characteristic 


Category 


Total Group 


Educational Level 






Eighth grade (8 years) 


1 


1.2 


Some high school (9-11 years) 


14 


17.5 


Completed high school [ic years; 


1Q 




Cnma mllanA 1 1 ^ v/Piirc) 
dome COIIc^ \ lO* 1 ycdl 9/ 


21 


26.2 


uompieieo couege \ id yeors; 


14 


17 S 


f^rarliiato Imnro than 1R V/Pflr^) 


10 


12.5 


uiner 


1 


1.2 


training ano experience 






No training or experience 


77 


96.2 


Some training and/or experience 


3 


3.8 


Current Occupational Status 






Student 


27 


33.8 


Military 


8 


10.0 


Employed-other 


20 


25.0 


Unemployed 


25 


31.2 



Approximately 82% of the sample was 30 years old or younger; 62% was 25 or 
younger, and approximately 27% was under 21. Only three students had prior 
programming experience or training; two of these students were computer operators and 
one had some training in the rudiments of computer programming, but not in the 
COBOL language. 

Nearly 99% of the sample had at least some high school education. Approximately 
81% had completed high school, and 57% had at least some college, while 31% had a 
college degree or beyond. 

While we consider the age spread of the sample to be appropriate, they were, if 
anything, somewhat overeducated for our purposes; the effect of their schooling on 
course performance will be shown in the results section of this report. 

While over half the sample had vision difficulties (generally a need for glasses), in 
response to an "exit questionnaire'' almost no one reported any problems .wading the 
course materials from the display devices. 

EXPERIMENTAL DESIGN 

As previously shown in Figure 1, the COBOL course was designed with a hierarchy 
of parts. It is important to note that the nature of this hierarchy is a set of partially 
ordered input-output relationships. The approach to derive the total set was task analysis, 
and is based on information processing requirements (e.g., see Merrill,^ or Seidel^). The 

^P.F. Merrill. Task Analysis- An Information Processing Approach, Tech Memo No, 27, Florida 
State University, Tallahassee, April 15, 1971. 
^Scidc!, 1971, op. ciL 
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course consisted of four divisions, comprising 33 modules or topics. Most of these 
modules were further subdivided into sections which included interactive questions 
requiring the student to make some kind of overt response. Figure 5 shows the relation- 
ship of course structure to course objectives. 




MO MO MO MO MO MO 




QO QO QO QO QO QO QO QO QO QO QO QO QO QO QO QO QO QO QO QO QO QO QO QO QO QO 



Where: CO = Course objective 

DO = Division objective 

MO = Module objective 

QO = Question objective 



Figure 5. Hierarchy of Objectives. 



The course structure was an outgrowth of aii instructional design permitting a 
learner control over his path through the subject matier. T-Sections (See Figure 1) were 
written so that general concepts were introduced first, and, as the student continued, the 
specifics were explained in detail. This technique enabled the student to discern, as early 
as possible, whether he thought he had sufficient mastery to skip to the Q-Section. It 
also enabled the student in REVIEW or RECAP mode to find the information he needed 
as early in the module as possible. 

The Q-Section was designed as a discrete, self-contained section including explicit 
directions to the student on the mechanics of answering the question(s) that tested the 
module objective. 

Authors found the course design and structure to be beneficial during the creation 
of materials, because it imposed enough constraints on the instruction to make different 
modules, written by different authors, compatible in form and general order of content 
presentation. It allowed, however, much individual freedom to authors in the creation of 
within-module strategies. The modules that used gaming as an instructional method met 
the strategy and course structure requirements, as did the more conventional instructional 
techniques employed in some of the other modules. 

The special features of the COBOL course were used as the experimental variables of 
this research. These four learner-control options were QUIZ, REVIEW, RECAP, and 
ROUTE. Prior to the start of the course, each student was randomly assigned to one of 
the 16 possible combinations of the learner-control options (see Table 2). Thus, some 
students had all options available, some had access to none, some had only one of the 
four, while others had a combination of two or three control options. 
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Table 2 



{nstructional Treatments 



option 
Combinations 




Learner Control Options 




ROUTE 


RECAP 


REVIEW 


QUIZ 


1 


y 


y 


y 


y 


2 


y 


y 


y 


n 


3 


y 


y 


n 


y 


4 


y 


y 


n 


n 


5 


y 


n 


y 


y 


6 


y 


n 


y 


n 


7 


y 


n 


n 


y 


8 


y 


n 


n 


n 


9 


n 


y 


y 


y 


1U 


n 


y 


y 


n 


11 


n 


y 


n 


y 


12 


n 


y 


n 


n 


13 


n 


n 


y 


y 


14 


n 


n 


y 


n 


15 


n 


n 


n 


y 


16 


n 


n 


n 


n 



y » option available 
n *^ option not available 



Every student saw the same topics, although not always in the same order since 
those who had ROUTE were able to modify the sequence to some extent. Further, the 
particular option combination to which students were assigned and the extent to which 
they exercised those options would cause varied exposure to a given topic 
among students. 

Students who failed the quiz for a given topic were branched back to the beginning 
of th-»v topic to restudy the material and retake those quiz items previously missed. The 
amount of time spent in restudy depended on the student's own discretion and on his 
option combination (e.g., a student with the QUIZ option could jump to it immediately, 
whereas a student without QUIZ would perforce see all instruction in the topic before his 
second attempt at the quiz items). Should the student fail the quiz a second time, the 
same procedure would be followed except that a staff member would monitor hi? third 
attempt at the quiz and clear up any misconceptions evidenced by the answers. 

At the end of each division of the course, the students were required to code, run, 
and debug a COBOL program to demonstrate mastery of the skills taught in the topics of 
that division. Successful completion of this task was a prerequisite to starting the topics 
of the next division. 
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Chapter 3 



RESULTS 



As our study was designed to meet the dual goal of instructional adequacy and 
demonstrate instructionally relevant treatment effects, the data were analyzed first for 
overall level of learning and, second, for experimental effects. The experimental analyses 
were accomplished on the total sample by means of analyses of variance (ANOVA) and 
on operationally defined subsamples of high and low performers by means of discrim- 
inant function analyses. The strength of effects was studied by multiple regres- 
sion techniques. 

EVIDENCE FOR LEARNING 

Figure 6 indicates that the great majority of the students did quite well in the 
course. Notice that the frequency distributions of objectives passed on first try for both 
Divisions A and B are severely tmncated with a marked positive skew. Seventy-five 
percent of the students achieved at least 70% of the objectives on the first try for 
Division A; fully half the students passed 75% of the objectives on the first try. The 
results for Division B are comparable. First quartile scores for Divisions A and B are 90% 
and 85%, respectively. Third quartile scores are identical— 70%. 

These data strongly suggested that the limited variability of student performance, 
coupled with a restricted range arid severely skewed distribution, would render a conven- 
tional technique such as the coi^lysis of variance inappropriate as a means of analyzing 
the effects of the independent variables. Such an occurrence is not uncommon in 
educational research where the instructional material used as a vehicle for experi- 
mentation must also meet the requirement that it teach well. Nevertheless, in order to see 
whether other dependent variables such as time or expectancies (e.g.. Level of Aspiration, 
LOA) might show treatment effects, we initially performed an analysis of variance on the 
total sample. 



TOTAL SAMPLE ANALYSES 

The dependent measures gathered in the experiment were initially analyzed by 
means of the analysis of variance. The analyses were computed on quiz scores for the 
first try only and on transit times for the first and second trits. Beyond these limits, 
most of the cells of the design would be either empty or of unequal size because of the 
great number of students who achieved mastery on their first or second attempts. 

The results of the analyses are presented in Table 3. Notice that the only statistically 
significant main effects (p < .05) occur for the units of instruction (divisions, topics). The 
very large F ratios obtained for all transit times, and the smaller but significant ones for 
quiz performance, compilation errors, and LOA indicate that the instructional units 
varied in their difficulty and length. 
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Figure 6. Frequency Distribution of Subjects, by Percentage of 
Learning Objectives Passed on First Try. 



Therefore, criterion scores (quiz dependent measures and compilation errors) were 
affected by course location (topic and division). The few significant interactions that 
involve the learner-control optioni> are due to the fact that the extent to which students 
make use of the options affects the amount of time spent on a topic. (See Experimental 
Design.) 



ERIC 



Z3 



25 



Table 3 



Analysis of Variance Results 



Dependent Measure 


Significant Main Effects 
& Interactions 


Ratio F 


Degrees of 
Freedom 


1 ransii i iiiie 








Telling/Practice 


Route X Quiz X Topics 


1.86 


20; 1280 


Section (1st try) 








Transit Timo 


1 opics 






Oui? Section 


Route X Topics 


2.07 


20; 1280 


(1st trv) 


Review X Topics 


1.60 


20;1 280 




Quiz X Topics 


1.68 


20;1280 




Route X Quiz X Topics 


2.07 


20;1280 


Total Transit Time 


Topics 


51.64 


20;1280 


All Sections 


Recap X Review 


4.01 


1;64 


(1st & 2nd tries) 


Review X Quiz 


4.84 


l;b4 


Transit Time 


Divisions 


41.41 


1;64 


Divisions 


Recap X Review 


4.01 


1;64 




Review X Quiz 


4.84 


1;64 


Quiz Score 


Topics 


1.75 


25;1600 


Compile Errors 


Divisions 


12.06 


1;64 


LOA 


Objective 


7.85 


21;1008 




Route X Recap 


5.41 


1;48 




Route X Objective 


2.48 


21;1008 




Recap X Quiz 


3.26 


1;48 




Route X Review X Quiz 


6.75 


1;48 




Review X Quiz X Objective 


1.55 


21;1008 



HIGH AND LOW PERFORMERS 

It was decided that, under the circumstances, a potentially productive way of 
investigating the effects of learner control would be through a comparison of the best 
and worst students in the sample for any treatment differences (e.g., in the use of the 
options). The "high/low" performer technique used in the development of psychological 
inventories was applied (Brennan* ). The individual items are analyzed for their capacity 
to discriminate between those whose overall test scores are high and those whose scores 
are low. In the present study we wanted to analyze the way learner-control options are 
used as discriminators of high and low performers in the COBOL course. 

The following questions were posed for this aspect of the analysis: 

(1) Who are the high performers in Division A of the course? Who are the 
low performers? 

(2) Who are the high performers in Division B? Who are the low performers? 



'R.L. Brcnnan. "Some Statistical Problems In the Evaluation of SclMnstmctional Programs,* 
Doctoral dissertation. Harvard University, University Miciufilms No. 70 23080, Ann Arbor, Michigan, 
1970. 
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(3) Do they differ in the way or manner in which they make use of 
the options? 

(4) Is there any difference in the attributes of the high and low performers 
that might help explain or predict differences in option use? 

Selection 

In order to have a sufficient number of observations for the analysis, it was decided 
that the 20 highest and lowest performers in each division would be identified for further 
study. A combination of absolute and relative performance criteria was used to select 
thein. The specific criteria for high and low performers in each division are shown in 
Table 4. 



Table 4 

Selection Criteria for High and Low Performers 

(Divisions A and B of the COBOL Course) 

High Performers on Division A 

Criterion 1 : Passed > 84.6% of first-attempt objectives. 

Criterion 2: Of first-attempt passed objectives, 72.7 % were > 100 percentile. 

Criterion 3: Top 20 ranked according to score 



/ ^ / obtained S-, \\ 
\ \max obtained Sj// 



Low Performers on Division A 

Criterion 1: Failed > 30.8% of first-attempt objectives. 

Criterion 2: Of failed first-attempt objectives, 75.0% were < 23 J5 percentile. 

Criterion 3: Bottom 20 ranked according to score 



/ ^ / obtained Sj \\ 
\ \ max obtained S; // 



H igh Performers on Division B 

Criterion 1: Passed > 84.6% of first-attempt objectives. 

Criterion 2: Of first-attempt passed objectives, 66.6% were > 98.75 percentile. 

Criterion 3: Top 20 ranked according to score 



/ „ / obtained S\ \\ 

score =S ( , . ' )) 

\ \max obtained S J/ 



Low Performers on Division B 

Criterion 1: Failed > 23.1% of first-attempt objectives. 

Criterion 2: Of first-attempt failed objectives, > 75.0% had scores < 26.25 percentile. 
Criterion 3: Bottom 20 ranked according to score 



/ ^ / obtained Sj \\ 
\ \max obtained S j/ 
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High Versus Low Performance: A Task-Dependent Phenomenon 

In an effort to identify the cognitive and affective characteristics on which high and 
low performers differ, the Entry Characteristics Test scores of these groups were 
subjected to multiple stepwise discriminant analysis. The results are summarized in 
Table 5. 

Table 5 



Summary of Stepwise Multiple Discriminant 
Analysis of High and Low Performers on 
35 Entry Characteristics Test Scores 







Standardized 


Division 


Test Selected 


Discriminant Weight 


A 


PMA-Verbai 


7.738 




IPAT Anxiety-Score "B" 


1.560 




Match Problems V 


2.506 




Significance of overall discrimination: 




F (3,36)= 10.154; p<.01 




B 


Cooperative English-Vocabulary 


19.742 




ATPP 


13.314 




PMA-Spatial Relations 


"6.57 




Letter Sets 


-2.693 




First and Last Names 


3.007 




(Associative Memory) 





Significance of overall discrimination: 
F (6.33)= 9.609; p<. 01 



The most striking set of findings is that there was no overlap in the ECTs 
discriminating high from low performers in Division A as opposed to Division B. (See 
Table 5.) In addition, there was virtually no overlap in the high and low performers in 
Division A, and those who were high and low performers respectively in Division B. (Six 
students were high in both divisions, four students were low in both.) The importance of 
these findings is that they emphasize the dependence of the phenomenon of high and low 
performance upon task-specific variables. That is, we have a task-by -individual char- 
acteristic relationship. Just because individuals may be high performers in one set of tasks 
or in one level of an instructional course does not guarantee that they will continue to be 
high performers in another portion of the course. Therefore, it is extremely important to 
consider the dynamic nature of the situation in which individuals interact with the 
instructional tasks. ' 

The overall discrimination for both divisions was highly significant (the overall F for 
Division A was 10.154; for Division B the F was 9.609; p < .01). In Division A, three of 
the 35 entry characteristics scores were selected by the analysis, the Primary Mental 
Abilities Test, a measure of general verbal aptitude. Match Problems, Test V, a measure of 
figural adaptive flexibility; and the IPAT Anxiety Test— Score B, a measure of the extent 
to which an individual reports anxiety -related feelings or behaviors. Examination of the 
standardized coefficients for these variables shows that the greater part of the 
discrimination is due to the verbal abilities test. The positive sign of the coefficients 
indicates that the high performers possess these attributes to a greater degree than the 
low performers. 

EMC 



For Division B, the analysis identified programming aptitude, the Aptitude Test for 
Programming Personnel, and vocabulary -measured by the Cooperative English Test, as 
the principal discriminators between the high and lov^ performers. Two tests— the PMA 
Spatial Relations and the Letter Sets Test (a measure of inductive reasoning ability)— are 
not easily interpreted in the present context. In relation to the nature of the instructional 
requirements, these capacities (factors) were hindrance:. The negative sign of the 
coefficients means the high performers possess fewer of these attributes than the low 
performers, although one would expect a positive relationship with success in the course. 
It is likely that the extremely small N, coupled with the particular statistical analysis, 
resulted in poswsible spurious results that masked valid relationships. 

Manner of Option Use 

The frequency with which the high and low performers used the options is 
presented in Table 6. The values have been adjusted to equate option availability across 
groups. This was necessary because, in some instances, the high and low performers did 
not come from treatment conditions having the same degree of access to options. 
(Because second attempts occurred with relatively low frequency, these data on option 
usage are not repoiled here.) Data on the ROUTE option usage are presented in Table 7. 



Table 6 



Frequency of First Attempt Option Usage 

(Adjusted) 





Options 


Performers 


RECAP 


REVIEW 


QUIZ 


Division A 










High 


6.2 


12.0 




15.0 


Low 


35.0 


65.0 




25.0 


Division B 










High 


15.0 


4.0 




8.8 


Low 


20.0 


52.5 




11.1 




Table 7 






Use of the ROUTE Option by 






High and Low Performers 




Performers 


ACTIVE 


PASSIVE 


TOTAL 


Division A 










High 


14 


5 




19 


Low 


10 


10 




20 


Division B 










High 


21 


11 




32 


Low 


18 


19 




37 
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Table 7 shows that in both divisions the low performers consistently used the 
options more frequently than the high performers. This suggests that, if the options aid 
learning at all, the gain is due not to how often they are used but rather to where and 
when they are used. 

Concerning the choice of ROUTE options, students who had the ROUTE option 
could, when presented with a "menu" of available topics, choose to pick their own or 
defer to the system to pick one at random. Table 6 shows that the ROUTE option was 
made available by the IDM far more often in Division B than in Division A; this occurs 
because the prerequisite structure in Division B is less ordered.* In both divisions, the 
high and low performers saw "menus" about the same number of times. However, the 
proportion of occurrences in which subjects made an active selection of the next topic 
differed markedly between the highs and lows. In Division A, the high performers made 
their own choice nearly three times more often than not; the low performers actively 
chose only half the time. In Division B, the high performers chose nearly twice as often 
as they deferred to the system; here, the low performers also chose about half the time. 

Another interesting finding is that when given an option other than a linear path, 
most of the high and low performers chose a different path. For example, in Division A 
(refer to Figure 3 for the prerequisite structure) where the students were permitted an 
option of going through F, G, H, and I as opposed to other alternatives, 11 out of 14 of 
the high performers chose another alternative than the linear. Of the low performers, (a 
total of 10) only one chose the linear path, whereas the others chose five different 
unique paths (e.g., G, F, H; A, F, H). Thus, path choice, while freely exercised did not 
differentiate between high and low performance. 

Expectancy Measure 

During the first IDM investigation in the current project, LOA (Level of Aspiration) 
was studied as a correlational variable. LOA correlated significantly with criterion 
performance in both levels of complexity in COBOLl. The correlations were +.42 and 
+.53 (p<.01), respectively. These findings were consistent with previous results in 
programmed instruction. (See ^^idel and Hunter.^ ) 

Because of the modularization of C0B0L2, it was possible to perform a finer-grain 
analysis than previously. The LOA dat? were analyzed at the level of specific objectives.^ 
The basic hypothesis test-d was that high performers would be more realistic than low 
performers. Operationally this would take the form of (a) a smaller positive discrepancy 
score for the high performers, and (b) a smaller absolute value of discrepancy between 
LOA and objective score for the high performers. 

A ceiling effect because of the excellent performance by the high performers 
prevented an analysis of the signed differences. However, the test of absolute value 
differences as shown in Figure 7 clearly supported the hypothesis that high performers 
would be more realistic than the low performers. Coupled with the previous LOA 
findings these results substantiate the value of providing an Expectancy Operator for 

^ See Figures 2 and 3. 

^Robert J. Seidel and Harold G. Hunger, The Application of Theoretical Factors in Teaching Problem 
Solving by Programmed Instruction, HumRRO Professional Paper 23-70, August 1970. 

^Because of the consistent descriptive findings, it had been decided to make LOA part of the 
decision-making strategy In the revised IDM for C0B0L2. Extensive course revision coupled with curtailed 
resources prevented implementation of LOA as the newly developed Expectancy Operator (a decision- 
making rule using the discrepancy between actual performance and anticipated performance by the 
learner), but LOA was used again as a correlational variable. The results were consistent with the previous 
data indicating significance of student expectations as predictors of achievement. 
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remediational purposes as part of an improved IDM. It is once again significant to note 
the consistency of the phenomenon across divisions. Generally, high performance v/as 
characterized by greater realism of expectancy than was low performance. Yet the 
specific individuals in whom the phenomenon was observed differed from Division A 
tasks to Division B tasks. 

Time Criteria: Differences Between High and Low Performers 

The first important finding, which had been expected, was that the high performers 
were significantly faster learners than the low performers. The means for completion time 
of Division A low and high performers are given in Table 8. 

Table 8 



Mean Completion Time (In Minutes) for 
High and Low Performers 



Performers 


Mean Completion 
Time 
(in minutes) 


f 


P 

(one>tailed) 


Division A 
High 


566.30 
(a=148) 


2.67 


<.025 


Low 


743.31 
(a = 257) 






Division B 
High 


494.94 
(a=119) 


1.69 


.05 


Low 


582.b9 
(a=199) 







Because of the significant differences between them, the high and low performers 
were analyzed separately by means of multiple-regression techniques. It may be assumed 
that they come from different populations. 

Because of the exploratory nature of the study, the small number of subjects, and 
the high degree of error variance in this kind of research environment, we also attempted 
to look at correlations with time criteria in a slightly different way. Simple Pearson r's 
were calculated separately for ECT scores and for Divisions A and B completion time. 
The findings to be presented in this section should, therefore, be viewed as hypothesis- 
generating, rather than hypothesis-confirming or conclusive statements. The findings are 
to be used as indicating potential relationships that require additional studies using a 
greater number of students. 

The results of the stepwise multiple-regression analyses for high and low performers 
in Division A are presented in Table 9. The high and low performers both showed as 
relationships between time to learn and convergent production of concepts, one of 
Guilford's SI factors, as well as programmer aptitude. The low performers also showed a 
relationship between general reasoning capability and associative memory with length of 
time to complete the course. 

In Division B (see Table 10) associative memory appears to be a desirable capacity as 
indicated by its inverse relationship with time of completion for high performers. 
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Table 9 



Results of Stepwise Multiple Regression Analyses:' 
Division A 



1 ^ 










Proportion 


Variable Entered j ^ 




/3 


t 


P 


Variance Predicted 



High Performers 

ATPP - 8.660 2.851 -.559 -3.038 <005 .580 

Sign Changes -10.658 5.990 -3.27 -1.779 <.05 .006 

Analysis of Variance for Regression: F (2,17) = 15.513 (p <.01) /? =« .8038 



Low Performers 



Sign Changes 


-19.034 


3.725 


-.6293 


-5.110 


<.005 


.728 


ATPP 


"18.947 


5.293 


-.6061 


-3.580 


<.005 


.082 


Ship Destination 


13.880 


6.273 


.3351 


2.213 


<.025 


.026 


First and Last Names 


11.758 


'4.190 


.3311 


2.806 


<.01 


.028 



Analysis of Variance for Regression: F (5,14) = 21.802 (p <.01) /? = .9414 



'Dependent Variable « Total Transit Time. Predictor Variables « 11 Entry Characteristic Tests. 



Table 10 

Results of Stepwise Multiple Regression Analyses:^ 
Division B 





A 


A 


A 






Proportion 


Variable Entered 


b 


S^b 




t 


P 


Variance Predicted 


High Performers 














Sentence Completion 


17.326 


3.257 


.7041 


5.320 


<.005 


.351 


First and Last Names 


- 7.543 


2.315 


-.4398 


-3.258 


<.005 


.316 


PMA Aptitude 


- 1.735 


.741 


-.334 


-2.341 


<.025 


.085 


Analysis of Variance for Regression: F (3,16) 


= 16.188 (P<.01) 






Low Performers 














Ship Destination 


-.16.093 


4.978 


-.5391 


-3.233 


<.005 


.588 


Sign Changes 


- 9.022 


3.858 


-.3899 


-2.339 


<.025 


.100 


Analysis of Variance for Regression: F (2,17) 


= 18.741 (p<.01) 







^Dependant Variable " Total Transit Time, Predictor Variables « 1 1 Entry Characteristic Tests 
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Another finding was that high performerb showed a relationship between sentence 
completion test scores and completion time. It is puzzling that this is a direct relationship 
indicating that the greater the achievement motivation, the more time the high 
performers seem to take to complete the instructional tasks in Division B. One could 
explain this if we assume that the high performers, given their high degree of motivation, 
were more careful and therefore took more time to ensure that they would achieve as 
much as possible. This is open to conjecture and requires further study. The low 
performers in Division B, as indicated in Table 10, showed relationships between learning 
time and general reasoning capability and convergent production of concepts. 

In addition to the stepwise multiple regression analyses, we examined the individual 
correlations between specific ECT scores and the time criteria for revisions A and B 
separately, and for high and low performers separately. 

In addition to programmer aptitude tests, the factors of logical reasoning and 
convergent production of concepts were significantly related to learning time in 
Division A for both high and low performers. The most important correlations unique to 
high performers were between the time criterion and the associative memory factor in 
Division A, and between the time criterion and the figural adaptive flexibility factor 
(word coding— capability of looking at materials in new ways). Table 11 shows that the 
correlation for the associative memory factor with the time criterion was - .52 (p< .05); 
the correlation of time and the figural adaptive flexibility factor was - .61 (p < .01). 

As seen in Table 11, the low performers had a different set of significant unique 
correlations with the time criterion in Division A. Here, the factor of general reasoning 
showed up again. This factor significantly contributed to the multiple correlation 
discussed previously— its Pearson r correlation was - .47. Also, the logical reasoning and 

Table 1 1 



Unique and Common^ Predictors of Division A Completion Time 
For High and Low Performers 

(Pearson r Correlations) 



Predictor 


Factor 


Correlations With Time^ 


High Performers 


Low Performers 


Common 








ATPP 




-.76** 


-.-/4** 


Sign Changes 


Convergent Production 


-.ey 


-.85"" 


PMA 






-.66** 


Inference Test 


Logical Reasoning 


.48* 


-.51* 


Unique 








Cooperative English 




-.39 


-.53* 


Ship Destination 


General Reasoning 


-.22 


-.47* 


Logical Reasoning 


Logical Reasoning (G) 


-.28 


-.49* 


Sentence Completion 


Achievement Motivation 


.10 


.52* 


First and Last Names 


Associative Memory 


-.52* 


-.29 


Word Coding 


Figural Adaptive Flexibility 


-.61** 


-.39 


Figure Classification 


Induction 


-.26 


-.33 



^Common predictors show correlations wah iime lot both hiijh and low performers, whose significarwe isp<!.05. 
^•Indicates statistical significance, p< 05. • * p<.01 , 
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achievement motivation factors showed significant relationships with the time criterion 
for the low performers (- .49 and +.52, respectively). 

In Division B (see Table 12), the unique predictors for the high performers seemed 
to be achievement motivation as was indicated by the multiple-regression analysis and 
associative memory. 

Table 12 

Unique and Common* Preciiotors of Division B Completion Time 
For High and Low Performers 



(Pearson r Correlations) 






Correlations With Tlme*^ 


Prfdictor 


Factor 


High Performers 


Low Performers 



Common 
None 

Unique 



Sign Changes 
Ship Destination 
PMA 
ATPP 

C-ooperative English 

ire nee Test 
Lujcal Reasoning 
Word Coding 
Sentence Completion 
First and Last Names 
Figure Classification 



Convergent Production 


-.38 


-.79* 


General Reasoning 


-.31 


-.77" 




-21 


-.70" 




-.25 


-.54" 




-.35 


-.54" 


Logical Reasoning 


-.16 


-.47* 


Logical Reasoning (G) 


-.002 


-.46* 


Figural Adaptive Flexibility 


-.26 


-.53* 


Achievement Motivation 


.59" 


.15 


Associative Memory 


-.56" 


-.12 


Induction 


.04 


-.18 



^Common predictors sho* cc-relations with time for both high and low performers whose significance is 
P<.05. 

indicates statistical significance, p < .Ob; **p<^ .01. 



The low performers in Division B, contrary to the results of the stepwise multiple- 
regression analysis, had many factors related to the time criteria. The convergent produc- 
tion of concepts and general reasoning factors were significantly related to the criterion 
as the multiple-regression analyses had shovm. However, marker tests used for logical 
reasoning were also significantly related to the criterion (• .47 for the inference test, -and 
-.46 for the logical reasoning test, p<.05 for both). Also significantly related to the 
criterion were the general characteristics of English aptitude and programmer aptitude. 
These were indicated by the PMA, ATPP, and Cooperative English Test. The correlations 
with the criterion for these tests were - .70, - .54, and - .54, respectively. 

In brief, the two different correlational analyses indicated that general programmer 
aptitude and the factor of convergent production of concepts are related to learning time 
in Division A of the course for both high and low performers. However, different sets of 
unique factors for high or low performers were found to be related to learning time in 
Division B. These findings provide further evidence of the importance of task variables, 
not only in the phenomenon of high versus low performance, but for learning time 
as well. 
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Chapter 4 
DISCUSSION AND IMPLICATIONS 



This research on learner control and on structure of intellect has resulted in a 
number of significant findings md implications. First, the study clearly developed an 
instructional vehicle which meets one of the criteria set down by Seidel^ concerning valid 
research in an instructional environment. Specifically, the data presented clearly showed 
that the students learned the COBOL course, and learned it well. 

Secondly, the study was performed in a rich instructional environment. Contrary to 
recent statements by Boutwell and Barton,^ the complex treatment task, as opposed to 
the factorially simple task, is preferred in performing Aptitude by Treatment Interaction 
(ATI) studies in a real instructional world. Many past difficulties in obtaining genei'- 
alizable and realistic results in instructional research havp come from the fact that rather 
simplistic approaches in a paired-associate setting have yielded findings inconsistent with 
findings in instructionally rich settings. 

The third significant aspect to the current study has been the development of a very 
useful way to characterize high and low performers with an operationally defined set of 
criteria. This has led us to an analytic approach different from the standard statistical 
inference techniques and has highlighted the value of discriminant function analyses in 
instructional research settings. Of great importance is the fact that the particular indi- 
viduals designated high or low performers differ depending upon the particular instruc- 
tional tasks involved (i.e., the phenomenon of high and low per{ormance was identifiable 
across two divisions of the COBOL course). High and low performers differed with 
respect to the usage of options, as well as their level of aspiration settings concerning 
their performance. But the particular individuals identified as high or low performers in 
Divisions A and B were quite different. There were overlaps of only 20-25%. This suggests 
that future researchers need to identify the instructional task characteristics related to 
student profiles of high and low performers. 

Another significant finding in our study was that self-assessment can make a 
significant contribution to instructional management, whether the latter is by students or 
by the learning system. The next step that should be taken is to use the instructional 
options based on expectations, as part of the decision-making process in an adaptive 
instructional environment. This approach is consistent with some of the comments made 
recently by Boutwell and Barton^ in dealing with the inconsistency of ATI research 
findings. They advocate searching for new variables as a basis for decision making in 
adaptive instructional settings. We agree with this and suggest an Expectamy Operator as 
one of these variables. It is clearly of a more dynamic nature than the standard aptitude 
tests which are designed to yield predictors within a relatively stable environment. We 
also suggest that the use of the Expectancy Operator is consistent with an approach to a 
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^Seidel, 1971, op. cit. 

^Richard C. Boutwell and Grant E. Barton. "Toward an Adaptive Learner Controlled Model of 
Instruction: A Place for the New Cognitive Aptitudes," Educational Technology, vol. 14, no. 5, May 
1974, pp. 13-18. 
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process-oriented model of decision making in which the student develops hi own 
heuristics toward making decisions, and applies the algorithm that he feels would be 
useful to solve problems. 



PROPOSED PRESCRIPTIVE USE OF THE EXPECTANCY OPERATOR 

The rationale behind the Expectancy Operator is based upon the relationship 
between a student's performance and the relative reality of his expectancies. If a student 
is judged to be unrealistic-with too great a discrepancy between LOA and perform- 
ance-a Probe path analysis and remediation are advocated. Recommended instructional 
guidelines for initial use of the Expectancy Operator and probe path foUow. 

It is suggested that implementation of the Expectancy Operator (LOA) take the 
form of providing for LOA measurement prior to a Q-Section or criterion test on specific 
subsets of objectives, and subsequent measurements of actual student performance for 
each subset of objectives. As part of a pilot implementation, there is the need to 
test-validate-the prescriptive value of LOA separately. Correlations do not ipso facto 
necessitate prescription, a causal interpretation of the relational outcome. 

The proposed IDM would take specific action based on a comparison of the 
preceding measurements. The action to be taken by the IDM is based upon the outcome 
of a problem in reality estimation or conceptual understanding, or some combination of 
these. In essence, the discrepancy score relationship between the LOA score in combina- 
tion with percent correct (and, when refined, confidence estimates) is to be used for 
determ:ning whether or not a Probe path should be followed. A Probe path is initiated 
by the system in order to gain more information concerning the student's problem and to 
take appropriate remedial action. 

For example, if, following the pretest, the alignment of expectation and percent 
correct represents "realistic" behavior, then the student is to follow the "normal" path 
for him. (Normality in this case is to be defined idiographically with a continually refined 
entrj' battery.) If discrepancies occur in the student's estimations of reality, then the 
Probe path will be followed for additional diagnosis and action (by noting the relation- 
ship between the measures of percent correct and the measures of the LOA). 

The specific plan and guideline for an initial implementation of an Expectancy 
Operator in a tutorial environment follow: 

1. Objective for using the Expectancy Operator: To lessen the relative distiince 
between Level of Aspiration and performance by raising the level of the 
lower (LOA or performance) to meet the hif;lier value. 

2. Plan for measuring LOA and providing solutions: 

a. Measure LOA prior to entering the Quiz. 

b. Measure performance in the Quiz. 

c. Feedback to student on LOA versus perfonnance, verbal plus numer- 
ical comparison. 

d. Provide solution. 

(1) Solution to raise performances to LOA: Follow-up alternate 
strategy with alternate quiz (check discrepancies). 

(2) Solution on subsequent modules to bring confidence level up to 
performance: Check discrepancy scores on subsequent modules. 

3. STATE DIAGNOSES: Eighteen possible states are derivable from the three 
sets of characteristics that follow. For purposes of initial implementation it 
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is suggested that A3a, A3b, Cla, Clb will be most useful as additions to an 
adaptive IDM. 

LOA PERFORMANCE ABILITIES 

A High 1 High a. Above Average 

B Average 2 Average b. Below/ Average 

C Low/ 3 Low/ 

4. Set of Alternative Instructional ACTIONS: 

a. Remedial modules geared to specific content failures concentrating on 
variation of practice exercises. 

b. Conference with proctor/instructor. 

c. Fun Option (on-line games). 

d. Skip ahead (practice). 

e. Leaving early. 

f. Alternate media module (CMI type, cassette, PI text). 

g. Compliment student. 

h. Do nothing. 

5. Recommended ACTIONS for DIAGNOSES (A3a, A3b, Cla, Clb): 

c, d, e, g — Confidence Building (Cla, Clb) for Low Expectation and 

High Performance, 
a, b, f Performance Building (A3a, A3b) for High Expectation and 

Low Performance. 

6. Actions to be implemented initially: 

STATE A3a: a 
STATE A3b: b and/or f 
STATE Cla: d, e, f, and/or g 
STATE Clb: g or h 

Probe Path Using Pre- and Post-Quiz Estimates 

General illustrations of how the Probe path would operate could also incorporate 
student estimates after his quiz performance. In previous research (e.g., Seidel and 
Hunter^ ), we have generally found with successful students that such a post-performance 
expectancy estimate is closer to actual performance than the LOA measure. Such an 
approach might be used as follows. First, let E stand for the student's estimate of his 
performance after the fact; let L stand for his Level of Aspiration or expectancy before 
the fact; and let A stand for his actual performance. The definition of i is trial number of 
a referent for the particular measurement number of LOA or EST. Applied to C0B0L2, 
it would reference module number. Generally, the values for L, E, and A would be 
derived from the degree of criterion attainment determined for a particular application in 
a given computer-based environment. 

A. DIAGNOSIS: If | Li - Ai | > | Ei - Ai | & (Ep-= +) 

Then student state is defined as REALISTIC to the IDM. 

ACTION: Diagnosis proceeds to next stage in IDM. Score on Test of Objec- 
tives, percent correct, defin6i> CONCEPTUAL state and confidence value defines 
REASSURANCE state. Given A, and passage of criteria here, student continues on 
"normal" module path available (based on current options—eventually to be redefined by 
our improved Entry Battery and better within-course historical predictors). Subject 
possesses the three R's— Realistic, Reassured, and Right (see Figure 8). 

^ Scidcl and Hunter, op. ciL ^H^cJ 
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Subject is REALISTIC, REASSURED, and RIGHT 



Yes 
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Figure R Illustration of Diagnostic States. 
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B. DIAGNOSIS: If I Lj- Ai |< I Ei 

ACTION: Go to PROBE path. 
PROBE path: Here IDM can be thought of in the following way. 

(1) It can query student directly to determine the nature of the problem as 
perceived by the student; that is, the student says, "No problem," or "I don't think I can 
hack it," or "I think I understand this stuff, but I'm not sure" (or some variation on 
this theme). 

(2) Diagnosis of problem is defined as the intersection of three orthogonal 
binary dimensions. The resulting state is estimated as follows (verbally below and 
pictorially in Figure 8): 

(a) Given B above (diagnosis of UNREALISTIC) and Low Confidence and 
High Objectives score, then two dimensional motivational problems exist— 
REASSURANCE and REAUSM. 

(b) Given B, and Low Confidence and Low Objectives score, then 
problem is diagnosed as both overall motivational and CONCEPTUAL. 

(c) Given B, High Confidence and High Objectives score, then problem is 
uniquely one of REALISM (e.g., the pessimist even though confident at time answering). 

(d) Given B, High Confidence and Low Objectives score, then also 
REALISM, but probably of different type (e.g., delusions of grandeur). 

(e) , (f), and (g): Cases currently handled by the IDM where the Expect- 
ancy Operator would be in the zero condition. 

Mathematically (and for IDM use) the state diagnosis can be described by ordered 
triples where 1 = a problem condition, a remedial operator is called for, and 0 = no 
problem. Thus, reading Realism, Reassurance, and Conceptual dimensions from left 
to right: 



Case (a) is described as < 1,1,0 > 
Case(b) < 1.1.1 > 
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Case (c) is described as < 1,0,1 > 
Case (d) < 1,0,0 > 

Cases (e), (f), and (g) < 0,1,0 > < 0,1,1 > < 0,0,1 > 



Use of Confidence Measure in Refined I DM 

A second measure of assessment used in our IDM research effort was confidence 
responding by the students. In COBOLl, unlike the implementation of LOA measures, 
confidence responding was part of every student response. The student gave an answer to 
a question and immediately distributed his confidence with respect to the answer over a 
series of alternatives if the alternatives wers available; or he attributed a degree of 
confidence to the correctness of the answer he provided in a completion type format. 

The results indicated a lowering of the correlation between confidence measures and 
correct responding as the student progressed through the 18 modules of COBOLl. The 
implementation scheme apparently was not a useful one for the students. They were 
required to give a percentage value between 0 and 100% using two digits as appropriate 
(e.g., 45, 55) and eventually adopted a principle of least effort. That is, the students 
either used a 0 or 100% confidence choice eventually, and the result was a lessening of 
the value of the confidence measure as an indicator of a state of under anding on the 
part of the student. 

The fact that the LOA measures indicated a high degree of value to self-assessment 
and the fact that other studies (e.g., Shuford, Albert, and Massengill^ ) supported the 
value of confidence measures as an aid to learning led us to re-evaluate the ways in which 
we would implement confidence measures in C0B0L2 (rather than eliminating confi- 
dence as a sensitive index of state of understanding). 

The goal in C0B0L2 was (a) to make the implementation easier for the student to 
use, (b) make all input responses equivalent in effort and difficulty, (c) lessen the 
frequency with which the confidence measures were used td avoid interrupting and 
interfering with the learning process, and (d) to increase the value of providing confidence 
measures by making associative materials attached to the various states of understanding 
more meaningful and positive than they had been for the student in COBOLl. 

The redesign was accomplished and initial off-line preliminary testing was achieved 
with staff members of the research project. However, because of the limited resources 
and other difficulties cited earlier, the re-implementation of confidence measures was not 
accomplished during this research project. We feel, nevertheless, that, in combination with 
the Expectancy Operator as discussed previously, the confidence measures should provide 
a very sensitive component to revised decision-making rules taking into account student 
motivation. The suggested implementation of confidence measures is provided as follows. 

Confidence testing would be part of the Q-Sections of the course, and they would 
be handled in the following manner. 

For the constructed response type of question, the student, after making sure that 
his answer is the one he wishes to have recorded and checked, will input his response. His 
display (CRT, hard copy, etc.) will be cleared and a confidence question will be 
displayed. This confidence question will summarize the task asked of the student and ask 
him to place his confidence in a prescribed location on the display. 

The student's confidence will be indicated by his selection of one of 11 characters 
from his keyboard. The characters are 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, and T-, where T stands 
for 10. A computer program computes the number of points the student receives by 

*E.H. Shuford, Jr., Arthur Albert, and H.E. Massenglll. "Admissible Probability Measurement 
Procedures," Psychometrikot vol. 31, no. 2» June 1966. 
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multiplying the number of points a question is worth (10-99), as determined by the 
author, by a three-place decimal associated with the student's confidence (see Table 13). 
For a correct response, the student receives that number of points. If he is incorrect, he 
receives the number of points found by multiplying the three-place decimal associated 
with the ten's complement of the student's confidence by the author's point value for 
the question. 

Table 13 



Scoring System for Comidence Measure 



Student Inputs for 
Correct Alternative 


Payoff 


0 


0.000 


1 


0.500 


2 


0.650 


3 


0.739 


4 


0.801 


5 


0.850 


6 


0.889 


7 


0.923 


8 


0.952 


9 


0.977 


T 


1.000 



For example, if the student places a 7 in his confidence block, then the ten's 
complement is taken as his "no confidence" response, in this case, 3. Suppose the author 
states that the question is worth 60 points; then, if the student is correct he receives 
60 X 0.923 = 55.3 rounded to 55 points, and if he is incorrect he receives 
60 X 0.739 = 44.3 rounded to 44 points. 

Each of the computer point values rolls onto the display in the proper location. If 
the student is dissatisfied with the number of points he will receive, he will change his 
confidence. The computation will be done again. It can be done as many times as the 
student wishes until he is satisfied with his potential number of points. He will then 
signal his completion by proper key press, and the appropriate number of points will be 
credited to him. 

The student will receive a feedback message on the display if his confidence is not 
one of the 11 characters just named. 

For multiple-choice questions, the student will distribute his confidence over all 
alternatives (as in COBOLl). His confidence must add to T (ten) and he must strike a 
character for each alternative. Once again, the weight of the question supplied by the 
author will be multiplied by the three-place decimal associated with the student's 
response. These products will be rolled onto the display in payoff fields next to each 
alternative. If the student is satisfied, he just presses the appropriate key. If he is not 
satisfied, he will change his confidences until he is happy with the payoff involved. When 
it is found that the student has pressed his key without changing his confidence 
assignments, he will be awarded the number of points he has assigned next to the 
correct alternative. 
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For example, if a question is worth 50 points, and the student distributes his 
confidence as follows on a four-alternative question whose second alternative is the 
correct one, he will receive 33 points: 



Payoff 

2 A. 33 Alternative 

2 B. 33 Alternative 

6 C. 44 Alternative 

0 D. 0 Alternative 



Here too, if the student types a character into his confidence that is not one of the 11 
characters mentioned, or if his confidences do not add to 10, he will receive feedback 
requesting him to correct his error in assigning his confidence to each of the alternatives. 



IMPLICATIONS OF EXPERIMENTAL FINDINGS FOR FUTURE RESEARCH 

Based on the research findings it is recommended that a combination of reality 
testing, confidence measuring, and conceptual responding be performed, to provide a 
useful baseline for a next generation instructional strategy (IDM). 

The results in earlier literature on learner control are generally conflicting, and the 
litest studies which the senior author reviewed (Jacobson and Thompson' ; McMullen^ ; 
and Judd, etal^) are consistent with this confusion. The problems in conflicting or 
unclear results stem from a number of ambiguities in the concept of learner control and 
its application to experimentation. The following discussion is an attempt to explicate 
these dimensions and indicate directions for research in order to determine the viability 
of the area generally labeled "learner controlled instruction." The dimensions to be 
considered are: level and/or type of instructional process, individual learner charac- 
teristics, instructional task, previous training or sophistication of student in self- 
management, and type and availability of learning resources. 

The first dimension is the level or type of instructional/educational process which is 
being studied. Significant differential effects may be revealed from the learner exercising 
a choice over what or how to study, depending upon whether we are speaking of a total 
curricular choice, a choice of some courses within a curriculum, objectives within a 
course, instructional options to reach prescribed objectives, time spent in all types of 
instruction, and so on. Obviously, there are different micro- or macro-levels of learner 
choice, and there are different complexities of the total educational/instructional process 
that are involved in such choices by students. 

The recent studies just cited outline these differences in bold relief: Jacobson and 
Thompson— rich but highly structured IPI instruction in math with learner control 
specified by a student deciding whether or not to follow directions on a sheet of paper; 
Judd, effl/. -^a short paired-associate task; McMullen**— a concept identification task with 



' Eric Jacobson and Murray Thompson. "Self-Managed Learning Using CAI," paper presented at the 
Annual Meeting of the American Educational Research Association, March 30 April 3, 1975, Washington, 

^ David W. McMuIIen. "Parallel Mastery With Learner Control," paper presented at the Annual 
Meeting of the American Educational Research Association, March 30'ApriI 3, 1975, Washington. 

^Wilson A. Judd, Kathleen Daubek, and Harold F. O'Neil, Jr. "Individual Differences in Learner 
Controlled CAI," paper presented at the Annual Meeting of the American Educational Research Associa- 
tion, March 30-April 3, 1975, Washington. 

^McMuUen, op. cit. 
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learner control over instance-selection, Luskin* -a college physics application where 
student-option consisted of decision to use or not to use a problem-solving algorithm 
following PI on its merits. Conclusions concerning the value of self-management or 
'ieamer control'* based upon one type or level of study may differ with evidence from 
another level or type. Unfortunately, there seems to be no literature that attempts to 
analyze such differences in making comparisons across levels or type of study. 

A second area for clarification is that of specific learner characteristics. Many (e.g., 
Atkinson^ ) do not consider the individual characteristics of prior learning history on the 
part of the student. Thus, to introduce a student who is well trained in the authoritarian 
mode of instructio*i to a small paired-associate task in a learner-controlled mode may 
defeat accurate assessment of the value of learner control. Obviously, prior history of the 
learner must be taken into account and need for re-training must be considered. 

Secondly, the most recent study by Judd and associates indicates a significant need 
for valid me^uring instruments of learning capability. Judd attempted to relate the 
effects of learner control to the personality construct of independence. The construct of 
independence was defined operationally by two tests, one that showed a significant 
relationship with criterion and one that did not. Judd's conclusion was that because one 
of the tests yielded statistical significance, the independence construct did show a 
relationship to learner control. However, one out of the two tests did not show a 
significant relationship with learner control. The real question, then, is: Is the construct 
validity or the measuring instruments suspect? It appears that before any conclusions can 
be drawn regarding the construct of independence and its relationship to learner control, 
more sensitive measuring instruments have to be studied. At the very least, a replication 
of Judd's study must be performed. 

With respect to other within-course history variables, the current research and the 
senior author's earlier work (Seidel and Hunter^ ) have shown the value of the dynamic 
properties of the student/task interaction for prediction of performance on criterion tests. 
The earlier study showed quite clearly that, with greater familiarity and skill in the 
instructional task, the student is able to be more realistic about his performance. 
Accordliigly, a reasonable hypothesis would be that the student, given this greater 
familiarity, should also be better able to determine what piece of instruction he should 
receive next. So far, our data on this point have been descriptive in nature (correlational). 
The next step would be to test out such hypotheses in a prescriptive manner by 
introducing Level of Aspiration into the decision -making process for instructional choice 

(e.g., as just noted). . , u • 

A third dimension requiring clarification is the kind of instructional task being 
studied under learner versus system-control conditions. It is quite clear that a taxonomy 
IS required. With the current state-of-the-art in instructional taxonomies, it does not niake 
any difference what taxonomy is chosen, so long as it is consistent (Hunter and Seidel, 
Ch. 4 in Hunter, etal^). One can certainly observe that some instructional tasks readily 
lend themselves to browsing or open-ended kinds of instruction, whereas other kinds do 
not. Also, operationally paired-associate learning is not the same as concept identification 
which, in turn, is not the same as problem-solving. 

* Bernard J. Luskin. "An Identification and Examination of Obstacles to the Development of Com- 
puter Assisted Instruction,** unpublished EdD Thesis, University of California, Los Angeles, 1970 

^Richard C. Atkinson. "Ingredients for a Theory of Instruction,** American Psychologist, vol 27, 
no. 10. October 1972. 

^Seidel and Hunter, op. ciL 
Beverly Hunter» Carol S. Kastncr, Martin L. Rubin» and Robert J. Seidel. Learning Alternatives in 
U.S. Education, Where Student and Computer Meet, Educational Technology Publications. Inc., 
Englewood Cliffs, New Jersey, 1975. 
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Thus, separate studies should be done to compare these various kinds of instruc- 
tional tasks relative to the learner-control variable before drawing any conclusions across 
instructional tasks. Also, the specific content to be learned is a related area that needs 
clarification vis a vis its relationship to learner control. Such a taxonomy of instructional 
tasks and instructional content permits concentrated study at a micro-level. At another 
more macro-level, one could study the effects of learner control over choice of instruc- 
tional task and/or content itself. McMullen's study, for example, showed no particular 
advantage to learner control within a comparison of serial versus parallel paradigms in 
concept learning. However, learner control over selection of paradigm has already been 
shown effective in concept learning and problem solving by Pask. 

A fourth dimension which must be clarified is the sophistication of the student in 
self-management, which is based upon previous, preferably long-term, training. This 
should be related to the first dimension discussed here— the type or level of the 
educational process being studied. Jacobson and Thompson's' recent study is one of the 
few attempts to provide training in self-management within a highly structured IPI 
environment, but confounding experimental variables prevent any firm conclusions. The 
question of transfer effects across levels or types of the instructional/educational process 
is in need of careful experimental study. 

Another dimension to consider is the type and availability of learner resources. It 
makes no sense to offer a student the option of controlling or managing his own learning 
process when the materials or resources are not available or amenable to browsing. For 
example, if we consider CAI, the entire TICCIT approach allowing a student to page back 
and forth through his materials is more amenable than a Coursewriter frame-oriented 
approach to CAI. 

Research on refinement of decision rules using these dimensions should aid further 
development of useful computer-based instructional materials in meaningful tuto- 
rial environments. 

Interpretation of the data in the current study can best 'be presented in terms of a 
provisional set of guidelines for prescribing instructional management. 

With respect to option control by students, the data suggest that we: 

(1) Establish high and low performer initial predictions for sets of similar 
instructional tasks. 

(2) Provide maximal student-control for the predicted high performers. 

(3) Design maximal system control for predicted low performers. 

(4) Track the performance of all. 

(5) Adjust the degree of student-control, based upon the changes in perform- 
ance, according to an empirical model that maintains the maximum percent 
of high performers. This model would have to be derived for each subject- 
matter application since tasks and instructional materials would have 
unique complexities and forms. 

The finding that high performers make more effective use of learner control options 
underlines the need and desirability for systematic study of the factors that may affect 
the utility of this feature for the student. If, for example, effective use of options 
depends on an accurate assessment by the student of his progress during instruction, then 
the appropriate research problem involves, (a) establishing the elements of information or 
data whose identification and evaluation underly the decision to use or not use a given 
option at a given time, (b) identifying the cognitive and attitudinal characteristics of 
learners that affect the accuracy of information processing and decision making, and 
(c) developing instructional strategies, which through careful empirical evaluation of these 



^Jacobson and Thompson, op. ciL 
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parameters, provide an amount and kind of control appropriate to each indi- 
vidual student. 

The ability to be able to discriminate, using an appropriate entry test battery, the 
high and low performers on an initial basis is related to the research problem. To the 
degree that this can be done, the designated student or system control options will be 
more or less appropriate. 

As noted from the preceding ECT analyses in the current study with a primarily 
verbal course like C0B0L2, the single best predictors were verbal entry tests such as the 
Primary Mental Abilities Test used for programmer aptitude selection, as well as the 
Cooperative English and, in another case, a uniquely suited Programmer Aptitude test 
called the ATPP. The significance of the ATPP, as well as the other structure of intellect 
tests which showed up as significant in our discriminant analysis, emphasizes that there 
will be other unique characteristics of any given instruction that will also aid in 
discriminating the predicted high from the low performers. These factors would have to 
be discerned from a structural analysis of the subject matter and its related tasks. (In the 
current instance, we are still doing analysis of this subject matter by factor structure 
using multiple raters to anive at a reliable index, vis-a-vis the Guilford Structure of 
Intellect characteristics.) 

Again, from the current study, supportive evidence for differential task transfer and 
the contribution of unique task and subject-matter characteristics comes from the 
comparison of Division A and Division B predictors from the ECT batteries. For example, 
in the introductory part of the course. Division A, the discriminant function was char- 
acterized by the most general and smallest number of predictor tests; however, in 
Division B, which was more heavily loaded with unique characteristics of COBOL 
programming and specific technological tasks, there were more variables present, and we 
found that the characteristics of these ECT predictors were more unique to the 
programming and specific tasks related to factor structure. Specifically, we refer to the 
fact that the ATPP, a uniquely oriented programmer aptitude test, was a heavily weighted 
factor under Division B prediction. Moreover, there was the appearance of Structure of 
Intellect factors involving associative memory and logical reasoning. In like manner, the 
anxiety test, IPAT B, was a predictor of performance in the introductory part of the 
course but dropped out as people became more familiar, comfortable, and sophisticated 
in the COBOL programming tasks. 

For longitudinal study and longer term transfer interpretation, it is necessary to test 
this hypothesis of specific task transfer with even more unique and specific, sophisticated 
COBOL course materials (like the other two divisions which were not available for our 
experimental subjects during the conduct of this ixisearch). It is predicted that more 
specific factors like logical reasoning and figural adaptive flexibility (the ability to change 
set with new materials) would take on even greater importance (relatively) to the specific 
nature of the task the individual would be encountering. This logic also applies equally to 
transfer across divisions within other hierarchically structured courses (e.g., electronics 
maintenance and other technical training). 

Combining the preceding discussion of option availability and relevant ECT for 
prediction of high or low performers with the previous description of the self assessment 
results provides an indication of a workable instructional decision model to be tested in 
future research. It would take something like the following form: 

Given that the high performer predicted by specific entry tests is more efficient 
in his use of options and is more realistic about his own performance, the self -assessment 
via the use of LOA could be used as a tracking device or technique for adjusting the 
degree of student control over the available remediational or accelerating options within 
the course of instruction. When a predicted low performer, for example, begins to fall 
within the realistic range of predicted high performers, that individual would then be 
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allowed more control over the use of available uptiorih. Note that the previous findings 
are descriptive (conelational) and Expectancy now ii> to be verified as a prescriptive 
variable in an IDM. 

However, when an individual falls outside the range of reality testing and is 
predicted to be a low performer, then the adjustment would take the form of eliminating 
student control over available options until such time as the individual performance 
begins to come more in line with reality and, indeed, until the predicted performance 
jumps back up to what a high performer would be. This model, however, does require 
continued research in order to verify its appropriateness to various applications of 
instructional tasks. 

As an adjunct to this model, it would also be relevant to add other parameters 
which describe in a more sensitive way the high or low performer*s state of under- 
standing. This might be done by use of the revised confidence measures discussed earlier 
under self-assessment. It may well be that a previously designated high performer who 
yields some unrealistic estimates of performance in some novel material might be 
signaling that he is getting into deep water and can no longer handle the instructional 
tasks required of him. In this case, supplementing the probing of that individual's 
understanding by the use of confidence techniques might provide additional indices for 
the kind of specific remediation unique to his requirements. All of the above awaits 
further verification in real-world instructional environment, similar to that used within 
the current study. 

Clearly, more research is required to get a closer identification of relevant ATI 
relationships in varied computer-based learning environments. 

Instructional management of resources (e.g., via the use of student-controlled 
options) must be considered relative to the student characteristics and still be relevant to 
the instructional tasks. While our results were not as definitive as we would have liked, 
differences in relevant ECT scores were found for high and low performers. Follow-up 
work should more carefully relate the entry tests to specific taxonomically labeled 
characteristics of the instructional tasks. It is challenging to consider within the same 
instructional environment, the differences in the sets of relationships revealed among ECT 
scores, high versus low performance, and learning time criteria. It is hoped that subse- 
quent research will reveal the reasons for these findingb and the ways in which they are 
related in a broader construct of learning. 
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